{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Get ICESat-2 Data \n",
    "#### Based on the Tutorial by Amy Steiker at ICESat-2 Hackweek, June 2019\n",
    "Modified by Kristin Poinar on Tuesday June 18 for #WaterNoice project\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 70,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "/home/jovyan/ATL06/seeps\n"
     ]
    },
    {
     "name": "stdin",
     "output_type": "stream",
     "text": [
      "Earthdata Login password:  ············\n"
     ]
    }
   ],
   "source": [
    "# Give path of polygon that outlines the area of interest for data download\n",
    "kml_filepath = str('../kmlfiles/seeps-koge.kml')\n",
    "kml_filepath = str('../kmlfiles/seeps.kml')\n",
    "\n",
    "# Give data set ID (e.g. ATL06) here, also known as \"short name\"\n",
    "short_name = 'ATL06'\n",
    "# short_name = 'ATL03'\n",
    "\n",
    "\n",
    "\n",
    "# Input start date in yyyy-MM-dd format\n",
    "start_date = '2018-10-22'\n",
    "# Input start time in HH:mm:ss format\n",
    "start_time = '00:00:00'\n",
    "# Input end date in yyyy-MM-dd format\n",
    "end_date = '2019-02-22'\n",
    "# Input end time in HH:mm:ss format\n",
    "end_time = '23:59:59'\n",
    "\n",
    "\n",
    "\n",
    "# Where to put the data?\n",
    "# Create an output folder if the folder does not already exist.\n",
    "\n",
    "path = str(os.getcwd() + '/SeepOutputs_' + kml_filepath[12:-4] + '_' + short_name)\n",
    "path = str('/home/jovyan/' + short_name + '/' + kml_filepath[12:-4])\n",
    "print(path)\n",
    "# if not os.path.exists(path):\n",
    "#     os.mkdir(path)\n",
    "\n",
    "# Earthdata Login credentials\n",
    "# Enter your Earthdata Login user name\n",
    "uid = 'kristinpoinar'\n",
    "# Enter your email address associated with your Earthdata Login account\n",
    "email = 'kpoinar@buffalo.edu'\n",
    "pswd = getpass.getpass('Earthdata Login password: ')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Import packages\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 71,
   "metadata": {},
   "outputs": [],
   "source": [
    "import requests\n",
    "import getpass\n",
    "import socket\n",
    "import json\n",
    "import zipfile\n",
    "import io\n",
    "import math\n",
    "import os\n",
    "import shutil\n",
    "import pprint\n",
    "import time\n",
    "import geopandas as gpd\n",
    "import matplotlib.pyplot as plt\n",
    "import fiona\n",
    "import h5py\n",
    "import re\n",
    "# To read KML files with geopandas, we will need to enable KML support in fiona (disabled by default)\n",
    "fiona.drvsupport.supported_drivers['LIBKML'] = 'rw'\n",
    "from shapely.geometry import Polygon, mapping\n",
    "from shapely.geometry.polygon import orient\n",
    "from statistics import mean\n",
    "from requests.auth import HTTPBasicAuth"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Create a token\n",
    "\n",
    "#### We will generate a token needed in order to access data using your Earthdata Login credentials, and we will apply that token to the following queries. If you do not already have an Earthdata Login account, go to http://urs.earthdata.nasa.gov to register. Your password will be prompted for privacy."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 72,
   "metadata": {
    "attributes": {
     "classes": [],
     "id": "",
     "n": "3"
    }
   },
   "outputs": [],
   "source": [
    "# Request token from Common Metadata Repository using Earthdata credentials\n",
    "token_api_url = 'https://cmr.earthdata.nasa.gov/legacy-services/rest/tokens'\n",
    "hostname = socket.gethostname()\n",
    "ip = socket.gethostbyname(hostname)\n",
    "\n",
    "data = {\n",
    "    'token': {\n",
    "        'username': uid,\n",
    "        'password': pswd,\n",
    "        'client_id': 'NSIDC_client_id',\n",
    "        'user_ip_address': ip\n",
    "    }\n",
    "}\n",
    "headers={'Accept': 'application/json'}\n",
    "response = requests.post(token_api_url, json=data, headers=headers)\n",
    "token = json.loads(response.content)['token']['id']\n",
    "#print(token)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Determine the number and size of granules available within a time range and location.\n",
    "\n",
    "#### Let's explore information about our data set. We'll start by determining the most recent version number of our data set. We will also find out how many data granules (files) exist over an area and time of interest. [The Common Metadata Repository](https://cmr.earthdata.nasa.gov/search/site/docs/search/api.html \"CMR API documentation\") is queried to explore this information. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 73,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "2018-10-22T00:00:00Z,2019-02-22T23:59:59Z\n"
     ]
    }
   ],
   "source": [
    "# Get json response from CMR collection metadata and print results. This provides high-level metadata on a data set or \"collection\", provide in json format.\n",
    "\n",
    "params = {\n",
    "    'short_name': short_name\n",
    "}\n",
    "\n",
    "cmr_collections_url = 'https://cmr.earthdata.nasa.gov/search/collections.json'\n",
    "response = requests.get(cmr_collections_url, params=params)\n",
    "results = json.loads(response.content)\n",
    "#pprint.pprint(results)\n",
    "\n",
    "# There may be cases where more than one data set version exists, which may happen when ICESat-2 data \n",
    "# version updates. Let's make sure we have the most recent version of our data set.\n",
    "\n",
    "# Find all instances of 'version_id' in metadata and print most recent version number\n",
    "\n",
    "versions = [i['version_id'] for i in results['feed']['entry']]\n",
    "latest_version = max(versions)\n",
    "#print(latest_version)\n",
    "\n",
    "# Parse temporal range \n",
    "temporal = start_date + 'T' + start_time + 'Z' + ',' + end_date + 'T' + end_time + 'Z'\n",
    "print(temporal)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Area of Interest input\n",
    "\n",
    "I'll be using Option 3, Spatial file input, including Esri Shapefile or KML/KMZ. \n",
    "Use geopandas to read in the file."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 74,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAeQAAAFlCAYAAAA6QpuEAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4xLjAsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy+17YcXAAAgAElEQVR4nO3deWyc933n8c+Xx1CiJIqkxEuHdYsUKVXU4UOW5UMyfaSSZcvRWq4TOFYSyXbrtLsIUqdeBF0sdpF2dxEYKNDASBsEC2/qNgfaRRG3si3bjWtHJnVYJ0mdvHVSvMVrfvsHh1xK5jGUZvg8M/N+AYOZeY6Z748Pf/zw98wzz2POOQEAAG8leV0AAAAgkAEA8AUCGQAAHyCQAQDwAQIZAAAfIJABAPCBlMl8s6ysLDdnzpzJfEsAACLqxIkTV5xzOZF+3UkN5Dlz5uidd96ZzLcEACCiVq1adSEar8suawAAfIBABgDABwhkAAB8gEAGAMAHCGQAAHyAQAYAwAcIZAAAfIBABgDABwhkAAB8gEAGAMAHCGQAAHyAQAYAwAcIZAAAfIBABgDABwhkAAB8gEAGAMAHxg1kMys0s8PDbq1m9ifD5n/XzJyZzY5uqQAAxK+U8RZwzlVKKpUkM0uWVC/p16Hn8yWVSaqJYo0AAMS9ie6y3iLpjHPuQuj5jyR9T5KLaFUAACSYiQbyLkk/lyQze0pSvXPuyFgrmNkeMys3s/Lm5ubbLBMAgPgWdiCbWUDSU5L+wczSJb0h6Qfjreece8s5t945tz4rK+v2KwUAII5NZIT8pKSDzrmLkpZIWiTpiJmdlzRP0kEzy498iQAAxL9xD+oa5nmFdlc7545Kyh2cEQrl9c65KxGtDgCABBHWCDm0i7pM0q+iWw4AAIkprBGyc65T0qwx5i+MVEEAACQiztQFAIAPEMgAAPgAgQwAgA8QyAAA+ACBDACADxDIAAD4AIEMAIAPEMgAAPgAgQwAgA8QyAAA+ACBDACADxDIAAD4AIEMAIAPEMgAAPgAgQwAgA8QyAAA+ACBDACADxDIAAD4AIEMAIAPEMgAAPgAgQwAgA8QyAAA+ACBDACADxDIAAD4AIEMAIAPEMgAAPgAgQwAgA8QyAAA+ACBDACADxDIAAD4AIEMAIAPEMgAAPgAgQwAgA8QyAAA+EDKeAuYWaGkd4ZNWizpB5LmStomqUfSGUkvOeeuR6NIAADi3bgjZOdcpXOu1DlXKmmdpE5Jv5a0T9JK59zvSaqS9P2oVgoAQByb6C7rLZLOOOcuOOf+1TnXF5r+maR5kS0NAIDEMdFA3iXp5yNM3y3pN3deDgAAiSnsQDazgKSnJP3DLdPfkNQn6e1R1ttjZuVmVt7c3HwntQIAELcmMkJ+UtJB59zFwQlm9qKkrZJecM65kVZyzr3lnFvvnFuflZV1Z9UCABCnxj3KepjnNWx3tZk9IelPJT3knOuMdGEAACSSsEbIZpYuqUzSr4ZN/itJMyTtM7PDZvbjKNQHAEBCCGuEHBoBz7pl2tKoVAQAQALiTF0AAPgAgQwAgA8QyAAA+ACBDACADxDIAAD4AIEMAIAPEMgAAPgAgQwAgA8QyAAA+ACBDACADxDIAAD4AIEMAIAPEMgAAPgAgQwAgA8QyAAA+ACBDACADxDIAAD4AIEMAIAPEMgAAPgAgQwAgA8QyAAA+ACBDACADxDIAAD4AIEMAIAPEMgAAPgAgQwAgA8QyAAA+ACBDACADxDIAAD4AIEMAIAPEMgAAPgAgQwAgA8QyAAA+ACBDACADxDIAAD4wLiBbGaFZnZ42K3VzP7EzLLNbJ+ZVYfusyajYAAA4tG4geycq3TOlTrnSiWtk9Qp6deSXpf0vnNumaT3Q88BAMBtmOgu6y2SzjjnLkjaLulnoek/k/R0JAsDACCRTDSQd0n6eehxnnOuUZJC97kjrWBme8ys3MzKm5ubb79SAADiWNiBbGYBSU9J+oeJvIFz7i3n3Hrn3PqsLD5mBgBgJBMZIT8p6aBz7mLo+UUzK5Ck0P2lSBcHAECimEggP6//v7takv5J0ouhxy9K+sdIFQUAQKIJK5DNLF1SmaRfDZv8Q0llZlYdmvfDyJcHAEBiSAlnIedcp6RZt0y7qoGjrgEAwB3iTF0AAPgAgQwAgA8QyAAA+ACBDACADxDIAAD4AIEMAIAPEMgAAPgAgQwAgA8QyAAA+ACBDACADxDIAAD4AIEMAIAPEMgAAPgAgQwAgA8QyAAA+ACBDACADxDIAAD4AIEMAIAPEMgAAPgAgQwAgA8QyAAA+ACBDACADxDIAAD4AIEMAIAPEMgAAPgAgQwAgA8QyAAA+ACBDACADxDIAAD4AIEMAIAPEMgAAPgAgQwAgA+keF0AAACxwDmn3t7eqL0+gQwAgKRgMKiuri51dnbedH/jxg11dnaqo6Mjqu8fViCbWaakn0haKclJ2i2pS9KPJU2R1CfpVefcgSjVCQBIEN3d3WpubpY0MCodNNrjcJcb/rivr09dXV1Dt8HwTU9PV0ZGhjIzM5WZmakFCxZo5syZQ7cpU6bojTfeiFxjhwl3hPympHedc181s4CkdEl/L+m/OOd+Y2ZfkfSXkh6OSpUAgLgWDAbV1NSk2tpaXbp0Sfn5+UpKSpKZSdKY94OPR5s+0n1qaqrmzp17U9jOmDFDycnJk9DakY0byGaWIelBSd+QJOdcj6QeM3OSMkKLzZTUEKUaAQBxyDmnlpYWXbhwQXV1dZo1a5bWrl2rkpISpaWleV3epAtnhLxY0mVJPzWz1ZIqJP2xpD+R9C9m9j81cLT2/VGrEgAQN27cuKGamhrV1dWpr69PpaWlevLJJ5Wdne11aZ4KJ5BTJK2V9Jpz7ndm9qak1zUwKv6Pzrlfmtl/kPQ3kh69dWUz2yNpjyQVFBRErHAAQOzo7+9XY2OjamtrdfXqVRUWFuqpp57SggULbtrlnMjs1g/Gv7SAWb6kz5xzC0PPN2kgkB+QlOmcczbw02xxzmWM/kpSSUmJe+eddyJSOADA35xzam5uHhoN5+Xlac2aNSouLlYgEPC6vNtmZhXOufWRft1xR8jOuSYzqzWzQudcpaQtkk5oYFf2Q5I+lLRZUnWkiwMAxJ6uri7V1NSotrZWZqbS0lJt3bpVmZmZXpfma+EeZf2apLdDR1iflfSSpH+U9KaZpUi6odBuaQBA4unv71dDQ4Nqa2t17do1rVixQjt27ND8+fPZJR2msALZOXdY0q3D899KWhfxigAAMcE5p6tXr6qmpkYNDQ2aM2eONmzYoKKiIqWmpnpdXszhTF0AgAlrbW1VeXm5zExr1qzR008/rYyMMQ8jwjgIZADAhNTU1Ojo0aMqKyvTmjVr2CUdIQQyACAs/f39+uKLL9Tc3KxvfOMbysvL87qkuEIgAwDG1d7ergMHDig/P1979+5NyDNpRRuBDAAYU319vQ4fPqxHHnlEd999N7uoo4RABgCMKBgM6tixY7p48aK+9rWvae7cuV6XFNcIZADAl3R2durAgQPKysrSyy+/rKlTp3pdUtwjkAEAN2lsbNShQ4e0ceNG3X///eyiniQEMgBA0sAu6pMnT6qurk67du3SXXfd5XVJCYVABgCoq6tL5eXlSk9P18svv6xp06Z5XVLCIZABIMFdunRJFRUVuvvuu/Xggw8qKSnJ65ISEoEMAAnKOadTp07pwoULevbZZ7V48WKvS0poBDIAJKDu7m6Vl5crNTVVe/fu1YwZM7wuKeERyACQYK5cuaLPP/9cpaWl2rJlC7uofYJABoAE4ZxTdXW1Tp8+re3bt6uwsNDrkjAMgQwACaCnp0cVFRVyzmnPnj3KzMz0uiTcgkAGgDh37do1ff755youLtZjjz2m5ORkr0vCCAhkAIhTzjmdPXtWlZWV2rp1q4qLi70uCWMgkAFETL8LqqmvU23BXs1KnqKs5DSlGAcMeaG3t1eHDh1Sd3e3vvWtbyk7O9vrkjAOAhnAbXPO6XqwRzW9barta1Ndb4d6FRyab5Iyk9I0K3nK0C0neaoykgPeFZ0AWlpadODAAS1ZskRPPvmkUlNTvS4JYSCQAUxId7BftX3tQyHcGuwddVknqTnYreZgt073tkiSplqyds8sVhIXLIiK8+fP6/jx43riiSe0evVqr8vBBBDIAMYUdE4X+ztV0zsQwhf7O+Xu4PW6XL/q+zo0P3V6xGqE1NfXpyNHjqi1tVW7d+9WTk6O1yVhgghkAF/S2t+jmr421fS2q66vXd2uP6Kvf6G3lUCOoNbWVh04cEDz58/X888/r0CAjwRiEYEM4CaHblzWb7sao/La+cnpWjNlthanzozK6yeSvr4+NTU1qaGhQZcuXVJZWZnWrl3LtYtjGIEM4CZtwZ6Ivp5JWpI6U2umzFZ+Cpf0uxO9vb1qbGxUY2OjLl26pLlz52r16tUqKiricolxgEAGcJPOYF9EXiegJJWkZWv1lNmakcQu1NvV3d09FMKXL1/WggULtG7dOhUWFmrq1Klel4cIIpAB3KTrDj8vzkgKaHXabBWnZSlgnBHqdty4cUMNDQ1qbGzUtWvXtHDhQt13331avny50tLSvC4PUUIgA7hJ122OkAtS0rUmLUeLUjP4StNt6OrqUn19vRobG3X9+nUtWbJEmzZt0tKlSzlIK0EQyABucs/UPHUGe9WroHpcUD2u/0v3vcPuF6ZmqHTKbOWlpHtdeszp6OhQfX29mpqa1NraquXLl2vz5s1asmSJUlL485xo2OIAbrI0wBHQ0dTW1ja0O7qjo0NFRUV67LHHtGjRIi76kOAIZACIstbW1qHd0d3d3SoqKtLv//7va8GCBUpK4lzfGEAgA0CEOefU0tIyFML9/f0qLi7W008/rXnz5hHCGBGBDAAR4JxTc3PzUAibmUpKSrRx40bNnTuXE3ZgXAQyANwm55yuXr2qhoYGNTQ0KBAIqKSkRA8//LDy8/MJYUwIgQwAExAMBnXlypWhA7PS09NVXFysRx99VDk5OYQwbltYgWxmmZJ+ImmlBq6otts596mZvSbpjyT1Sfpn59z3olYpAHgkGAzq0qVLamxsVENDg2bOnKmSkhI9+eSTmjVrltflIU6EO0J+U9K7zrmvmllAUrqZPSJpu6Tfc851m1lu1KoEgEnW39+vixcvDp22Mjs7WytXrtTWrVuVlZXldXmIQ+MGspllSHpQ0jckyTnXI6nHzF6R9EPnXHdo+qUo1gkAUdfX16eLFy+qoaFBTU1Nys3N1cqVK/XMM88oIyPD6/IQ58IZIS+WdFnST81staQKSX8sabmkTWb23yTdkPRd59znt65sZnsk7ZGkgoKCSNUNAGNyzqm/v1+9vb1j3gaX6enp0dWrV1VQUKBVq1Zp586dmj6dazZj8oQTyCmS1kp6zTn3OzN7U9LroelZku6TdLekvzezxc45N3xl59xbkt6SpJKSkpvmAcCd6O3tVUtLi65fv662tja1traqu7tbPT096u3tVVJSkgKBgNLS0pSWlqYpU6YoLS1NU6dO1YwZM4amDd7mz5+v9HROAQpvhBPIdZLqnHO/Cz3/hQYCuU7Sr0IBfMDMgpJma2A0DQAR45xTZ2enrl+/rpaWFrW1ten69eu6ceOGZs+erfz8fBUVFSkvL0/Tp08fClpORYlYMm4gO+eazKzWzAqdc5WStkg6IemMpM2SPjSz5ZICkq5EtVoAca+/v18tLS1Dt7a2NjU3NysQCCgvL08FBQUqKSlRXl6esrOzOesV4ka4R1m/Junt0BHWZyW9JKlD0t+a2TFJPZJevHV3NQCMxjmnGzduDAVva2urWlpa1N7eruzsbOXl5Wnx4sUqKChQXl4eu5IR98IKZOfcYUnrR5j1tciWAyAeBYPBod3Mw3c5S1Jubq4KCgq0dOlS5efna/bs2exqRkLiTF0AIsY5p66uLrW2tg4dZDV4mzFjhvLz8zV//nzl5+crPz9f06dP58xWQAiBDGDCnHPq6OgYCt729vahAA4EApo1a9bQd3jz8/OVm5ur1NRUr8sGfI1ABjCqYDCo9vb2LwVvW1ub0tPTNXv2bOXm5mrRokXKyclRTk6OpkyZ4nXZQEwikAGor69vxOBtb29XRkbGUPAWFhZq9uzZmj17tgKBgNdlA3GFQAYSSG9v79Cu5cHAbW1tVVdXlzIzM5WTkzO0qzknJ0fZ2dlKSeHPBDAZ6GlAnAoGg6qpqVFLS4s6OjrU0tKinp4eZWdnKycnR/PmzRvazZyVlcX3eQGPEchAHGpvb1d5ebmmT5+u5cuXDwXvzJkzOaoZ8CkCGYgzFy5c0LFjx/TQQw/p3nvvJYCBGEEgA3Git7dXR44cUVtbm1588UXl5+d7XRKACSCQgThw9epVlZeXa9myZXrhhRf4zi8QgwhkIIY551RZWamzZ89q69atKi4u9rokALeJQAZiVGdnpyoqKpSWlqa9e/dq5syZXpcE4A4QyEAMqq+v15EjR3Tvvfdq06ZNfGUJiAMEMhBD+vr6dOzYMV2+fFnPP/+85s+f73VJACKEQAZiREtLiz7//HPNnTtXr7zyCueMBuIMgQz4nHNOZ86cUWVlpR5//HGtXr2a7xYDcYhABnysu7tbBw8eVH9/v771rW9p1qxZXpcEIEoIZMCnLl68qIMHD2r16tXasmWLkpOTvS4JQBQRyIDPBINBnThxQnV1dXr22We1ePFir0sCMAkIZMBnjh49qv7+fr3yyiuaNm2a1+UAmCR8eRHwmRs3bmj9+vWEMZBgCGTAZ5xzfF4MJCACGfCZYDDImbeABESvB3yGETKQmAhkwGcYIQOJiV4P+EwwGGSEDCQgAhnwGeccI2QgAdHrAZ9hhAwkJgIZ8Bk+QwYSE70e8BmOsgYSE4EM+AwjZCAx0esBn+EzZCAxEciAzzBCBhITvR7wGUbIQGIKK5DNLNPMfmFmp8zspJltGDbvu2bmzGx29MoEEkNfX5/6+/sJZCABhXs95Dclveuc+6qZBSSlS5KZzZdUJqkmSvUBCcM5p8OHD2vFihUKBAJelwNgko07QjazDEkPSvobSXLO9Tjnrodm/0jS9yS5qFUIJIhz586po6NDTz31lNelAPBAOLusF0u6LOmnZnbIzH5iZtPM7ClJ9c65I9EtEYh/zc3NOnnypHbt2qXU1FSvywHggXACOUXSWkl/7ZxbI6lD0p9LekPSD8Zb2cz2mFm5mZU3NzffSa1AXOrp6dGBAwe0bds2zZo1y+tyAHgknECuk1TnnPtd6PkvNBDQiyQdMbPzkuZJOmhm+beu7Jx7yzm33jm3PisrK0JlA/HBOaeKigqVlJSouLjY63IAeGjcQHbONUmqNbPC0KQtkg4653Kdcwudcws1ENprQ8sCCFNlZaWSkpJUVlbmdSkAPBbuUdavSXo7dIT1WUkvRa8kIDFcvHhR58+f1969e/maE4DwAtk5d1jS+jHmL4xUQUAi6OzsVEVFhXbu3KkZM2Z4XQ4AH+BMXcAkCwaD+vzzz7VhwwYtWrTI63IA+ASBDEyyo0ePKjMzUw888IDXpQDwEQIZmES1tbW6fPmyduzYITPzuhwAPkIgA5OktbVVX3zxhZ577jlNnTrV63IA+AyBDEyCvr4+HThwQGVlZSooKPC6HAA+FO7XngDcpt7eXh06dEgLFizQ2rVrvS4HgE8RyECU9Pb26syZMzpz5owWL16srVu3el0SAB8jkIEI6+np0enTp3Xu3DktXbpU3/zmNzV7NpcLBzA2AhmIkO7u7qEgLiws1Le//W1lZ2d7XRaAGEEgA3foxo0bOn36tM6fP6/i4mLt3btXXEgFwEQRyMBt6urqUnV1tWpqarRy5Uq98sormjlzptdlAYhRBDIwQZ2dnaqurlZtba1Wr16tbdu2cT5qAHeMQAbC1NHRoaqqKtXX12vNmjXavn27pk+f7nVZAOIEgQyMo729XVVVVWpoaNC6deu0Y8cOTZs2zeuyAMQZAhkYRVtbm6qqqtTU1KS7775bO3fu5JSXAKKGQAZu0draqsrKSl26dEn33XefnnvuOU2ZMsXrsgDEOQIZCLl+/bqqqqp05coVbdiwQX/wB3+gtLQ0r8sCkCAIZCS85uZmVVZWqrm5WRs3btT69esVCAS8LgtAgiGQkbCuXbumyspKtbS06IEHHtC6deuUmprqdVkAEhSBjIRz5coVVVZWqqOjQ5s2bdKaNWuUkkJXAOAt/gohITjnhoK4q6tLDz74oEpLS5WcnOx1aQAgiUBGnHPO6dKlS6qsrFRvb68eeughrVq1iiAG4DsEMuKSc05NTU2qqqpSMBjUQw89pJUrVyopKcnr0gBgRAQy4lJjY6MqKiq0bds2FRcXE8QAfI+/UohLubm5Sk5OVk5ODmEMICbwlwpxKSUlRcuWLdP777/vdSkAEBYCGXFr8eLFamhoUH19vdelAMC4CGTEreTkZC1fvpxRMoCYMKmB3NbWpvb29sl8SyS4hQsX6vLly7pw4YLXpQDAmCY1kKdNm6aPPvqIXYiYNElJSSosLNR7770n55zX5QDAqCY9kL/+9a/r8OHD6ujomMy3RgK766671NraqnPnznldCgCMatI/Q54zZ44eeOABHTx4kBELJkVSUpKKiooYJQPwNU8O6tqwYYMCgYBOnz7txdsjAc2bN09dXV2qrq72uhQAGJEngZyUlKQdO3aoqqpKra2tXpSABGNmjJIB+JpnX3vKysrSli1bVFFRoWAw6FUZSCBz5sxRf3+/Tp486XUpAPAlYQWymWWa2S/M7JSZnTSzDWb2P0LPvzCzX5tZ5kTffN26dcrMzFRlZeXEKwcmyMy0YsUKffDBB/wTCMB3wh0hvynpXedckaTVkk5K2idppXPu9yRVSfr+RN/czLR9+3adO3dOzc3NE10dmLC8vDwlJSXp2LFjXpcCADcZN5DNLEPSg5L+RpKccz3OuevOuX91zvWFFvtM0rzbKSAjI0NPPPGEKioq1N/ffzsvAYRtcJS8f/9+ft8A+Eo4I+TFki5L+qmZHTKzn5jZtFuW2S3pNyOtbGZ7zKzczMovX7484husWrVKBQUFOnHixERqB25LTk6O0tLSdOTIEa9LAYAh4QRyiqS1kv7aObdGUoek1wdnmtkbkvokvT3Sys65t5xz651z63NyckZ8AzPTtm3bVF9frytXrky0DcCErVixQh9++KH6+vrGXxgAJkE4gVwnqc4597vQ819oIKBlZi9K2irpBXeH3yVJT0/X1q1bdfDgQf5IIuqys7MlSXV1dR5XAgADxg1k51yTpFozKwxN2iLphJk9IelPJT3lnOuMRDFFRUVatGiRjh49GomXA0bV1NSk1NRU3XXXXV6XAgCSwj/K+jVJb5vZF5JKJf13SX8laYakfWZ22Mx+HImCvvKVr+jy5ctqamqKxMsBX+Kc04kTJ1RWVqakJK5ACsAfUsJZyDl3WNL6WyYvjXw5Ulpamp555hn98pe/1ObNmxUIBKLxNkhgFy5c0PTp01VYWDj+wgAwSXw5PFi0aJGKi4s5ChYR19/fr1OnTunxxx+XmXldDgAM8WUgS1JZWZna2to46AYRdfr0ac2bN0/z5t3W1+YBIGp8G8ipqanasWOHvvjiC924ccPrchAHuru7dfr0aZWVlXldCgB8iW8DWRq4ZN7atWt16NAhrtCDO1ZZWani4mLNmjXL61IA4Et8HciS9Mgjj6i3t1cXLlzwuhTEsI6ODtXU1OiRRx7xuhQAGJHvAzk5OVnPPvusjh8/ro6ODq/LQYw6efKk7r33Xk2fPt3rUgBgRL4PZGngCj3333+/Dh06xGXzMGHXr1/X5cuXdf/993tdCgCMKiYCWZI2btyomTNn6v3331dDQwOfKSNsx48f18MPP6y0tDSvSwGAUcVMICclJen555/Xtm3bVFVVpU8++UTXr1/3uiz43MWLF9Xd3a21a9d6XQoAjClmAnnQ0qVL9eqrr2rdunX69NNPdfDgQXV1dXldFnzIOafjx4/r0UcfVXJystflAMCYYi6QpYHR8t13363XXntN+fn5+uCDD3Tq1CkuOI+b1NbWasqUKVqxYoXXpQDAuGIykAdNmTJFjz32mL797W+rr69P+/btU01NDZ8vQ/39/Tp58iSnyAQQM2I6kAdlZ2dr165d2rlzp2pqavTxxx/r6tWrXpcFD505c0b5+flasGCB16UAQFjiIpAHLViwQHv37tXGjRtVXl6uAwcO8N3lBNTT06Pq6mo99thjXpcCAGEL6/KLscTMVFpaquLiYn3yySfav3+/Fi5cqMLCQqWmpnpdHiZBdXW1ioqKlJOT43UpABC2uBohDxcIBPTII4/o1VdfVSAQ0L59+3Tu3Dk+X45zXV1dOnfuHKfIBBBz4m6EfKuMjAzt2LFDDQ0N+s1vfqOzZ89q5cqVysvL87o0REFVVZVKS0uVkZHhdSkAMCFxO0K+1Zw5c7R7926VlZXp6NGj+vTTT9XW1uZ1WYigjo4O1dbWatOmTV6XAgATFvcj5OHMTMXFxVq+fLk+++wzffzxx5o3b54yMzNvWmak9cZ6Ptq02113rGXS09O5QMIoTp06pXvuuUfTpk3zuhQAmLCECuRBKSkpeuCBB7RmzRp98sknamtru+mz5bEejzYvnGUi8fjKlSsqKChQUVGRpk6dGkZrE0Nra6suXryoXbt2eV0KANyWhAzkQdOmTYu5r8Z0dXXp448/1vvvv68lS5Zo2bJlSklJ6M0oaeDyihs3buQCEgBiVsJ8hhwvpk6dqscff1wvv/yyJHH0uKRr166publZ99xzj9elAMBtY2gVozIzM7Vz507V19fr3Xff1dmzZ1VSUqK8vLyEO1XkyZMn9fDDD/M9cwAxjRFyjJs7d652796txx9/XCdOnNC///u/J9RlKS9duqSuri6tWbPG61IA4I4wQo4DZqaioiItW7ZMFRUV+uijj5Sbm6sVK1YoPT3d6/KixjmnkydPavPmzVxeEUDMI5DjSHJysu655x6tXr1a//Zv/6YPPvhAixYt0vLly+Nyd25jY6MkadWqVR5XAgB3jl3WcSgtLU2PPvqoXnnlFbi5QG8AAAfBSURBVKWmpmrfvn06c+aMgsGg16VFzODoeMuWLQn3mTmA+MQIOY7NnDlTO3bsUFNT000HfhUUFMR8iNXV1Wnq1KkqLCz0uhQAiAhGyAkgPz9fL774orZu3aqqqir99re/1bVr17wu67YFg0GdPHlSZWVlMf+PBQAMYoScIMxMy5Yt05IlS3To0CHt379fs2bNUnFxccydavL8+fPKzs7WokWLvC4FACKGEXKCSUpK0rp16/Sd73xHCxcu1P79+3Xs2DH19PR4XVpY+vv7VVVVpbKyMq9LAYCIIpAT1OD1ov/wD/9QU6dO1b59+1RdXe37A7/OnDmjuXPnau7cuV6XAgARRSAnuBkzZmj79u166aWX1N7ervfee091dXW+PBVnb2+vqqurtWXLFq9LAYCII5AhScrNzdXXv/51Pf300zp37pw+/vhjXb161euybnL69GktXbpUubm5XpcCABEXViCbWaaZ/cLMTpnZSTPbYGbZZrbPzKpD91nRLhbRt3jxYr388svauHGjysvLdeDAAbW3t3tdlrq7u3XmzBlt3rzZ61IAICrCPcr6TUnvOue+amYBSemS/kzS+865H5rZ65Jel/SnUaoTk8jMVFpaqpKSEn366af66KOPNH/+fBUWFk7o8obOOQWDQfX19am/v/+m+9EeD19+8NbX16euri6tXLlSWVn83wcgPtl4nxWaWYakI5IWu2ELm1mlpIedc41mViDpQ+fcmGdpWL9+vSsvL49A2ZhMHR0d2r9/v44fP6558+bJzG4KzOHBeuvNzJSamvqlWyAQGLoPBAJKS0sbenzr/MH73NxczlkNwHNmVuGcWx/p1w1nhLxY0mVJPzWz1ZIqJP2xpDznXKMkhUKZD/bi1LRp07R161bdd999OnHihFJSUkYMzJGmJSVxmAIAhCOcQE6RtFbSa86535nZmxrYPR0WM9sjaU/oabeZHZt4mb43W9IVr4uIknhtG+2KPfHaNtoVe6Jyzt5wdlnnS/rMObcw9HyTBgJ5qSa4y9rMyqMxzPdavLZLit+20a7YE69to12xJ1ptG3d/onOuSVKtmQ2G7RZJJyT9k6QXQ9NelPSPkS4OAIBEEe5R1q9Jejt0hPVZSS9pIMz/3sy+KalG0s7olAgAQPwLK5Cdc4cljTQ8n+gpk96a4PKxIl7bJcVv22hX7InXttGu2BOVto37GTIAAIg+vpMCAIAPRDyQzezPzazezA6Hbl8ZNu/7ZnbazCrN7PFR1vf1KTnN7Ltm5sxsduj5C8PaetjMgmZWOsJ6o/5c/GCEdi00s65h9f54lPV8vb2kEdtWZmYVZnY0dD/i+ThjbZuFpsVsHzOz/2pmX4R+1v9qZnNC02O+j43RtpjuZ2O0K+b72GhtC82LTj9zzkX0JunPJX13hOnFGjjjV5qkRZLOSEoeYbm/lPR66PHrkv4i0jXeQdvmS/oXSRckzR5h/ipJZyfyc/HDbaR2SVoo6VgY6/p2e43RtjWS5oQer5RUHyfbLKb7mKSMYY+/I+nHIywTq31sxLbFej8bo13x0MdGa1vU+tlk7rLeLunvnHPdzrlzkk5LumeU5X4WevwzSU9PUn3h+JGk70ka7YP35yX9fPLKiZjx2jUWP28vaYS2OecOOecaQk+PS5piZuGfpNsfRtpmMd3HnHOtw55O08i/jzHZx8Js21hiapvFQx8bY5tFrZ9FK5D/KDTU/9thw/S5kmqHLVMXmnarm07JKckXp+Q0s6c08F/ekTEWe05j/7EY6efiqXHatcjMDpnZRzZwQpiR+HJ7SWFvs2clHXLOdY8yP5a2WUz3MUkys/9mZrWSXpD0gxEWibk+NmiMtsV6Pxtvm8VcHxs0Stui1s9uK5DN7D0zOzbCbbukv5a0RFKppEZJ/2twtRFeyleHeI/Trjc08i/b4Lr3Sup0zo12atDRfi5Rd5vtapR0l3NujaT/JOn/2MCFRnzlDrdZiaS/kLR3lEVibZvFeh+Tc+4N59x8SW9L+qNb1vVtH5Nuu22+72d3uM1828ek225b9PpZlPfBL1To8xFJ35f0/WHz/kXShhHWqZRUEHpcIKnSB58lrJJ0SdL50K1PAydDyR+2zI8k/dlEfy5+b9ewZT+UtD4Wtlc4bZM0T1KVpI3xss1iuY+NUOOCW3/esdjHwm3bsHkx1c/Gales9rHx2hbNfhaNo6wLhj19RtLgf7P/JGmXmaWZ2SJJyyQdGOElfHdKTufcUedcrnNuoRs4p3edpLVu4LSiMrMkDZyp7O9Ge40xfi6eGatdZpZjZsmSZGaLNbC9zo7wMr7bXtK4bcuU9M8a6FSfjPYasbbNFMN9TJLMbNmwp09JOjVsXkz2sUGjtS3W+9kY7YrZPjZojN/H6PWzKPwn8b8lHZX0RaiggmHz3tDAEWmVkp4cNv0nCv1XKGmWpPclVYfus73+72iENp7XsKOsJT2sgQtw3Lrc8HaN+nPxy214uzTwuc9xDRxNeFDStljdXiO07T9L6pB0eNgtN9a3Weh5zPYxSb/UwB/kLyT9X0lzh82L6T42WttivZ+N0a6Y72Pj/D5GpZ9xpi4AAHyAM3UBAOADBDIAAD5AIAMA4AMEMgAAPkAgAwDgAwQyAAA+QCADAOADBDIAAD7w/wAy6nPcG+oU0QAAAABJRU5ErkJggg==\n",
      "text/plain": [
       "<Figure size 864x432 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "# Use geopandas to read in polygon file\n",
    "# Note: a shapefile or geojson, or almost any other vector-based spatial data format could be substituted here.\n",
    "\n",
    "# kml_filepath = str('../kmlfiles/seeps-koge.kml')\n",
    "# kml_filepath = str('../kmlfiles/seeps.kml')\n",
    "\n",
    "#Return a GeoDataFrame object\n",
    "gdf = gpd.read_file(kml_filepath)\n",
    "gdf.head()\n",
    "\n",
    "# And visualize it\n",
    "%matplotlib inline\n",
    "\n",
    "# Load \"Natural Earth” countries dataset, bundled with GeoPandas\n",
    "world = gpd.read_file(gpd.datasets.get_path('naturalearth_lowres'))\n",
    "\n",
    "# Overlay glacier outline\n",
    "f, ax = plt.subplots(1, figsize=(12, 6))\n",
    "world.plot(ax=ax, facecolor='lightgray', edgecolor='gray')\n",
    "gdf.plot(ax=ax, cmap='Set2')\n",
    "ax.set_ylim([60, 75])\n",
    "ax.set_xlim([-50,-30]);\n",
    "\n",
    "# Simplify and reorder the GeoDataFrame object using the shapely package \n",
    "# and convert the object back to a dictionary to be applied to the CMR polygon parameter. \n",
    "# Simplification is needed in order to pass a reasonable request length to CMR. \n",
    "# You may need to modify the simplification tolerance depending on the number of points of your polygon.\n",
    "\n",
    "#Integer position based indexing of GeoDataFrame object to get it into a shapeply geometry object.\n",
    "poly = gdf.iloc[0].geometry\n",
    "\n",
    "# Simplify polygon. The larger the tolerance value, the more simplified the polygon.\n",
    "#poly = poly.simplify(0, preserve_topology=False)\n",
    "\n",
    "# Orient counter-clockwise\n",
    "poly = orient(poly, sign=1.0)\n",
    "\n",
    "#Format dictionary to polygon coordinate pairs for CMR polygon filtering\n",
    "polygon = ','.join([str(c) for xy in zip(*poly.exterior.coords.xy) for c in xy])\n",
    "\n",
    "# aoi value used for CMR params below\n",
    "aoi = '3'"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### We will now populate dictionaries to be applied to our search query below based on spatial and temporal inputs. For additional search parameters, see the [The Common Metadata Repository API documentation](https://cmr.earthdata.nasa.gov/search/site/docs/search/api.html \"CMR API documentation\").\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 75,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "1191.8033456802"
      ]
     },
     "execution_count": 75,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "#Create CMR parameters used for granule search. Modify params depending on bounding_box or polygon input.\n",
    "\n",
    "if aoi == '1':\n",
    "# bounding box input:\n",
    "    params = {\n",
    "    'short_name': short_name,\n",
    "    'version': latest_version,\n",
    "    'temporal': temporal,\n",
    "    'page_size': 100,\n",
    "    'page_num': 1,\n",
    "    'bounding_box': bounding_box\n",
    "    }\n",
    "else:\n",
    "    \n",
    "# If polygon input (either via coordinate pairs or shapefile/KML/KMZ):\n",
    "    params = {\n",
    "    'short_name': short_name,\n",
    "    'version': latest_version,\n",
    "    'temporal': temporal,\n",
    "    'page_size': 100,\n",
    "    'page_num': 1,\n",
    "    'polygon': polygon,\n",
    "    }\n",
    "\n",
    "#print('CMR search parameters: ', params)\n",
    "\n",
    "\n",
    "\n",
    "# Query number of granules using our (paging over results)\n",
    "\n",
    "granule_search_url = 'https://cmr.earthdata.nasa.gov/search/granules'\n",
    "\n",
    "granules = []\n",
    "while True:\n",
    "    response = requests.get(granule_search_url, params=params, headers=headers)\n",
    "    results = json.loads(response.content)\n",
    "\n",
    "    if len(results['feed']['entry']) == 0:\n",
    "        # Out of results, so break out of loop\n",
    "        break\n",
    "\n",
    "    # Collect results and increment page_num\n",
    "    granules.extend(results['feed']['entry'])\n",
    "    params['page_num'] += 1\n",
    "\n",
    "    \n",
    "# Get number of granules over my area and time of interest\n",
    "len(granules)\n",
    "\n",
    "granule_sizes = [float(granule['granule_size']) for granule in granules]\n",
    "\n",
    "# Average size of granules in MB\n",
    "mean(granule_sizes)\n",
    "\n",
    "# Total volume in MB\n",
    "sum(granule_sizes)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Select the subsetting and reformatting services enabled for your data set of interest.\n",
    "The NSIDC DAAC supports customization services on many of our NASA Earthdata mission collections. Reformatting and subsetting are available on all Level-2 and -3 ICESat-2 data sets. Let's discover the specific service options supported for this data set and select which of these services we want to request. \n",
    "\n",
    "We will start by querying the service capability to gather and select customization options."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 76,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[{'id': 'ICESAT2', 'spatialSubsetting': 'true', 'spatialSubsettingShapefile': 'true', 'temporalSubsetting': 'true', 'type': 'both', 'maxGransSyncRequest': '100', 'maxGransAsyncRequest': '2000'}]\n"
     ]
    }
   ],
   "source": [
    "# Query service capability URL \n",
    "\n",
    "from xml.etree import ElementTree as ET\n",
    "\n",
    "capability_url = f'https://n5eil02u.ecs.nsidc.org/egi/capabilities/{short_name}.{latest_version}.xml'\n",
    "\n",
    "#print(capability_url)\n",
    "\n",
    "# Create session to store cookie and pass credentials to capabilities url\n",
    "\n",
    "session = requests.session()\n",
    "s = session.get(capability_url)\n",
    "response = session.get(s.url,auth=(uid,pswd))\n",
    "\n",
    "root = ET.fromstring(response.content)\n",
    "\n",
    "\n",
    "# collect lists with each service option\n",
    "\n",
    "subagent = [subset_agent.attrib for subset_agent in root.iter('SubsetAgent')]\n",
    "\n",
    "# variable subsetting\n",
    "variables = [SubsetVariable.attrib for SubsetVariable in root.iter('SubsetVariable')]  \n",
    "variables_raw = [variables[i]['value'] for i in range(len(variables))]\n",
    "variables_join = [''.join(('/',v)) if v.startswith('/') == False else v for v in variables_raw] \n",
    "variable_vals = [v.replace(':', '/') for v in variables_join]\n",
    "\n",
    "# reformatting\n",
    "formats = [Format.attrib for Format in root.iter('Format')]\n",
    "format_vals = [formats[i]['value'] for i in range(len(formats))]\n",
    "format_vals.remove('')\n",
    "\n",
    "# reprojection only applicable on ICESat-2 L3B products, yet to be available. \n",
    "\n",
    "# reformatting options that support reprojection\n",
    "normalproj = [Projections.attrib for Projections in root.iter('Projections')]\n",
    "normalproj_vals = []\n",
    "normalproj_vals.append(normalproj[0]['normalProj'])\n",
    "format_proj = normalproj_vals[0].split(',')\n",
    "format_proj.remove('')\n",
    "format_proj.append('No reformatting')\n",
    "\n",
    "#reprojection options\n",
    "projections = [Projection.attrib for Projection in root.iter('Projection')]\n",
    "proj_vals = []\n",
    "for i in range(len(projections)):\n",
    "    if (projections[i]['value']) != 'NO_CHANGE' :\n",
    "        proj_vals.append(projections[i]['value'])\n",
    "        \n",
    "# reformatting options that do not support reprojection\n",
    "no_proj = [i for i in format_vals if i not in format_proj]\n",
    "\n",
    "\n",
    "# Let's confirm that subset services exist for our data set by reviewing the `subagent` list. \n",
    "# If the list contains service information, we know that services are available. If not, we need to \n",
    "# set the `agent` API parameter to `NO` to indicate that our request will bypass the subsetter. \n",
    "# This will quickly send back the data \"natively\" without any customization applied.\n",
    "print(subagent)\n",
    "if len(subagent) < 1 :\n",
    "    agent = 'NO'"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### We'll begin populating the subsetting and reformatting parameters used for our NSIDC API request. In addition to the CMR information we queried above, the NSIDC API accepts Key-Value-Pairs (KVPs) for subsetting and reformatting services.\n",
    "\n",
    "#### Let's start with spatial subsetting. Recall that there are three options to *filter* our search results by spatial constraint: \n",
    "\n",
    "1) Bounding Box: Corresponding to the CMR `bounding_box` KVP\n",
    "\n",
    "2) Polygon coordinate pairs: Corresponding to the CMR `polygon` KVP\n",
    "\n",
    "3) Spatial file input, including Esri Shapefile or KML/KMZ: We simplified the file input to also be read by the CMR `polygon` KVP \n",
    "    \n",
    "#### We see above that `spatialSubsetting` is `true` and `spatialSubsettingShapefile` is `true`. Therefore the same *filtering* options can be applied to our *subset* constraint, with unique KVPs for the subsetting service:\n",
    "\n",
    "1) Bounding Box: `bbox` subset KVP\n",
    "\n",
    "2) Polygon coordinate pairs: `bounding_shape` subset KVP in [GeoJSON](https://geojson.org/) format. \n",
    "\n",
    "3) Spatial file input: The file can be read directly by the subsetter without simplification. This file will be posted to the API endpoint, so we don't need to specify an additional subset KVP here. \n",
    "\n",
    "#### Temporal subsetting is next, since we saw above that `temporalSubsetting` is `true`. We filtered data over 22 Feb 2019 and we can also subset the data to those dates if desired. \n",
    "\n",
    "The `time` KVP is used to subset temporally. This can be entered in the following formats:\n",
    "\n",
    "`time=yyyy-mm-dd,yyyy-mm-dd`\n",
    "\n",
    "`time=yyy-mm-ddThh:MM:ss,yyy-mm-ddThh:MM:ss` "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 77,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "2018-10-22T00:00:00,2019-02-22T23:59:59\n"
     ]
    }
   ],
   "source": [
    "# Temporal subsetting KVP\n",
    "\n",
    "timevar = start_date + 'T' + start_time + ',' + end_date + 'T' + end_time\n",
    "print(timevar)\n",
    "\n",
    "# Finally, let's determine if variable subsetting is available by finding the \n",
    "# length of the `variable_vals` list we gathered from the capabilities URL. \n",
    "len(variable_vals)\n",
    "\n",
    "\n",
    "# And we can enter a list of variables to subset separated by comma using the `coverage` key. \n",
    "# All forward slashes need to be included to indicate HDF group hierarchy.\n",
    "\n",
    "coverage = '/ancillary_data/atlas_sdp_gps_epoch,\\\n",
    "/gt1l/land_ice_segments/atl06_quality_summary,\\\n",
    "/gt1l/land_ice_segments/delta_time,\\\n",
    "/gt1l/land_ice_segments/h_li,\\\n",
    "/gt1l/land_ice_segments/h_li_sigma,\\\n",
    "/gt1l/land_ice_segments/latitude,\\\n",
    "/gt1l/land_ice_segments/longitude,\\\n",
    "/gt1l/land_ice_segments/segment_id,\\\n",
    "/gt1l/land_ice_segments/sigma_geo_h,\\\n",
    "/gt1r/land_ice_segments/atl06_quality_summary,\\\n",
    "/gt1r/land_ice_segments/delta_time,\\\n",
    "/gt1r/land_ice_segments/h_li,\\\n",
    "/gt1r/land_ice_segments/h_li_sigma,\\\n",
    "/gt1r/land_ice_segments/latitude,\\\n",
    "/gt1r/land_ice_segments/longitude,\\\n",
    "/gt1r/land_ice_segments/segment_id,\\\n",
    "/gt1r/land_ice_segments/sigma_geo_h,\\\n",
    "/gt2l/land_ice_segments/atl06_quality_summary,\\\n",
    "/gt2l/land_ice_segments/delta_time,\\\n",
    "/gt2l/land_ice_segments/h_li,\\\n",
    "/gt2l/land_ice_segments/h_li_sigma,\\\n",
    "/gt2l/land_ice_segments/latitude,\\\n",
    "/gt2l/land_ice_segments/longitude,\\\n",
    "/gt2l/land_ice_segments/segment_id,\\\n",
    "/gt2l/land_ice_segments/sigma_geo_h,\\\n",
    "/gt2r/land_ice_segments/atl06_quality_summary,\\\n",
    "/gt2r/land_ice_segments/delta_time,\\\n",
    "/gt2r/land_ice_segments/h_li,\\\n",
    "/gt2r/land_ice_segments/h_li_sigma,\\\n",
    "/gt2r/land_ice_segments/latitude,\\\n",
    "/gt2r/land_ice_segments/longitude,\\\n",
    "/gt2r/land_ice_segments/segment_id,\\\n",
    "/gt2r/land_ice_segments/sigma_geo_h,\\\n",
    "/gt3l/land_ice_segments/atl06_quality_summary,\\\n",
    "/gt3l/land_ice_segments/delta_time,\\\n",
    "/gt3l/land_ice_segments/h_li,\\\n",
    "/gt3l/land_ice_segments/h_li_sigma,\\\n",
    "/gt3l/land_ice_segments/latitude,\\\n",
    "/gt3l/land_ice_segments/longitude,\\\n",
    "/gt3l/land_ice_segments/segment_id,\\\n",
    "/gt3l/land_ice_segments/sigma_geo_h,\\\n",
    "/gt3r/land_ice_segments/atl06_quality_summary,\\\n",
    "/gt3r/land_ice_segments/delta_time,\\\n",
    "/gt3r/land_ice_segments/h_li,\\\n",
    "/gt3r/land_ice_segments/h_li_sigma,\\\n",
    "/gt3r/land_ice_segments/latitude,\\\n",
    "/gt3r/land_ice_segments/longitude,\\\n",
    "/gt3r/land_ice_segments/segment_id,\\\n",
    "/gt3r/land_ice_segments/sigma_geo_h,\\\n",
    "/orbit_info/cycle_number,\\\n",
    "/orbit_info/rgt,\\\n",
    "/orbit_info/orbit_number' "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Request data from the NSIDC data access API.\n",
    "\n",
    "#### Let's go over the configuration parameters:\n",
    "\n",
    "* `request_mode`\n",
    "* `page_size`\n",
    "* `page_num`\n",
    "\n",
    "`request_mode` is \"synchronous\" by default, meaning that the request relies on a direct, continous connection between you and the API endpoint. Outputs are directly downloaded, or \"streamed\" to your working directory. For this tutorial, we will set the request mode to asynchronous, which will allow concurrent requests to be queued and processed without the need for a continuous connection.\n",
    "\n",
    "**Use the streaming `request_mode` with caution: While it can be beneficial to stream outputs directly to your local directory, note that timeout errors can result depending on the size of the request, and your request will not be queued in the system if NSIDC is experiencing high request volume. For best performance, I recommend setting `page_size=1` to download individual outputs, which will eliminate extra time needed to zip outputs and will ensure faster processing times per request. An example streaming request loop is available at the bottom of the tutorial below. **\n",
    "\n",
    "Recall that we queried the total number and volume of granules prior to applying customization services. `page_size` and `page_num` can be used to adjust the number of granules per request up to a limit of 2000 granules for asynchronous, and 100 granules for synchronous (streaming). For now, let's select 10 granules to be processed in each zipped request. For ATL06, the granule size can exceed 100 MB so we want to choose a granule count that provides us with a reasonable zipped download size. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 78,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "2\n"
     ]
    }
   ],
   "source": [
    "#Set NSIDC data access base URL\n",
    "base_url = 'https://n5eil02u.ecs.nsidc.org/egi/request'\n",
    "\n",
    "# Set number of granules requested per order, which we will initially set to 10.\n",
    "page_size = 10\n",
    "\n",
    "#Determine number of pages basd on page_size and total granules. Loop requests by this value\n",
    "page_num = math.ceil(len(granules)/page_size)\n",
    "\n",
    "#Set request mode. \n",
    "request_mode = 'async'\n",
    "\n",
    "# Determine how many individual orders we will request based on the number of granules requested\n",
    "\n",
    "print(page_num)\n",
    "\n",
    "\n",
    "# We'll also create a new dictionary of NSIDC API KVPs to be used in our subset request. \n",
    "# Because we are looping through each page of requests, we'll add the `page_num` KVP to our dictionary \n",
    "# within the loop below.\n",
    "\n",
    "#Print API base URL + request parameters\n",
    "API_request = f'{base_url}?short_name={short_name}&version={latest_version}&temporal={temporal}&time={timevar}&polygon={polygon}&Coverage={coverage}&request_mode={request_mode}&page_size={page_size}&page_num={page_num}&token={token}&email={email}'\n",
    "#print(API_request)\n",
    "\n",
    "subset_params = {\n",
    "    'short_name': short_name, \n",
    "    'version': latest_version, \n",
    "    'temporal': temporal, \n",
    "    'time': timevar, \n",
    "    'polygon': polygon,  \n",
    "    'request_mode': request_mode, \n",
    "    'page_size': page_size,  \n",
    "    'token': token, \n",
    "    'email': email, \n",
    "    'Coverage': coverage,\n",
    "    }\n",
    "# print(subset_params)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### After all of these KVP inputs, what does our request look like? Here's a summary of all possible KVPs that we explored, both for CMR searching and for the subsetter:\n",
    "\n",
    "#### CMR search keys:\n",
    "* `short_name=`\n",
    "* `version=`\n",
    "* `temporal=`\n",
    "* `bounding_box=`\n",
    "* `polygon=`\n",
    "\n",
    "#### Customization service keys:\n",
    "* `time=`\n",
    "* `bbox=`\n",
    "* `bounding_shape=` \n",
    "* `format=`\n",
    "* `projection=`\n",
    "* `projection_parameters=`\n",
    "* `Coverage=`\n",
    "\n",
    "#### No customization (access only):\n",
    "* `agent=`    \n",
    "* `include_meta=` \n",
    "    * `Y` by default. `N` for No metadata requested.\n",
    "\n",
    "#### Request configuration keys:\n",
    "* `request_mode=` \n",
    "* `page_size=`\n",
    "* `page_num=`\n",
    "* `token=`\n",
    "* `email=`\n",
    "\n",
    "\n",
    "#### We'll request the same data but without any subsetting services applied. Let's create another request parameter dictionary with the `time` and `coverage` service keys removed, and we'll add `agent=NO` instead."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Request Data"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 79,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Order:  1\n",
      "Request HTTP response:  201\n",
      "Order request URL:  https://n5eil02u.ecs.nsidc.org/egi/request?short_name=ATL06&version=001&temporal=2018-10-22T00%3A00%3A00Z%2C2019-02-22T23%3A59%3A59Z&time=2018-10-22T00%3A00%3A00%2C2019-02-22T23%3A59%3A59&polygon=-38.4473901024492%2C66.9086039587665%2C-39.2106092081328%2C66.6988972341601%2C-39.231182022375%2C66.4596776379638%2C-38.6939296965283%2C66.4309364993326%2C-38.0544246374648%2C66.7145224687654%2C-38.0989491581698%2C66.8478161818577%2C-38.4473901024492%2C66.9086039587665&request_mode=async&page_size=10&token=EB953953-2F0A-785C-19A8-99256CBDDFDE&email=kpoinar%40buffalo.edu&Coverage=%2Fancillary_data%2Fatlas_sdp_gps_epoch%2C%2Fgt1l%2Fland_ice_segments%2Fatl06_quality_summary%2C%2Fgt1l%2Fland_ice_segments%2Fdelta_time%2C%2Fgt1l%2Fland_ice_segments%2Fh_li%2C%2Fgt1l%2Fland_ice_segments%2Fh_li_sigma%2C%2Fgt1l%2Fland_ice_segments%2Flatitude%2C%2Fgt1l%2Fland_ice_segments%2Flongitude%2C%2Fgt1l%2Fland_ice_segments%2Fsegment_id%2C%2Fgt1l%2Fland_ice_segments%2Fsigma_geo_h%2C%2Fgt1r%2Fland_ice_segments%2Fatl06_quality_summary%2C%2Fgt1r%2Fland_ice_segments%2Fdelta_time%2C%2Fgt1r%2Fland_ice_segments%2Fh_li%2C%2Fgt1r%2Fland_ice_segments%2Fh_li_sigma%2C%2Fgt1r%2Fland_ice_segments%2Flatitude%2C%2Fgt1r%2Fland_ice_segments%2Flongitude%2C%2Fgt1r%2Fland_ice_segments%2Fsegment_id%2C%2Fgt1r%2Fland_ice_segments%2Fsigma_geo_h%2C%2Fgt2l%2Fland_ice_segments%2Fatl06_quality_summary%2C%2Fgt2l%2Fland_ice_segments%2Fdelta_time%2C%2Fgt2l%2Fland_ice_segments%2Fh_li%2C%2Fgt2l%2Fland_ice_segments%2Fh_li_sigma%2C%2Fgt2l%2Fland_ice_segments%2Flatitude%2C%2Fgt2l%2Fland_ice_segments%2Flongitude%2C%2Fgt2l%2Fland_ice_segments%2Fsegment_id%2C%2Fgt2l%2Fland_ice_segments%2Fsigma_geo_h%2C%2Fgt2r%2Fland_ice_segments%2Fatl06_quality_summary%2C%2Fgt2r%2Fland_ice_segments%2Fdelta_time%2C%2Fgt2r%2Fland_ice_segments%2Fh_li%2C%2Fgt2r%2Fland_ice_segments%2Fh_li_sigma%2C%2Fgt2r%2Fland_ice_segments%2Flatitude%2C%2Fgt2r%2Fland_ice_segments%2Flongitude%2C%2Fgt2r%2Fland_ice_segments%2Fsegment_id%2C%2Fgt2r%2Fland_ice_segments%2Fsigma_geo_h%2C%2Fgt3l%2Fland_ice_segments%2Fatl06_quality_summary%2C%2Fgt3l%2Fland_ice_segments%2Fdelta_time%2C%2Fgt3l%2Fland_ice_segments%2Fh_li%2C%2Fgt3l%2Fland_ice_segments%2Fh_li_sigma%2C%2Fgt3l%2Fland_ice_segments%2Flatitude%2C%2Fgt3l%2Fland_ice_segments%2Flongitude%2C%2Fgt3l%2Fland_ice_segments%2Fsegment_id%2C%2Fgt3l%2Fland_ice_segments%2Fsigma_geo_h%2C%2Fgt3r%2Fland_ice_segments%2Fatl06_quality_summary%2C%2Fgt3r%2Fland_ice_segments%2Fdelta_time%2C%2Fgt3r%2Fland_ice_segments%2Fh_li%2C%2Fgt3r%2Fland_ice_segments%2Fh_li_sigma%2C%2Fgt3r%2Fland_ice_segments%2Flatitude%2C%2Fgt3r%2Fland_ice_segments%2Flongitude%2C%2Fgt3r%2Fland_ice_segments%2Fsegment_id%2C%2Fgt3r%2Fland_ice_segments%2Fsigma_geo_h%2C%2Forbit_info%2Fcycle_number%2C%2Forbit_info%2Frgt%2C%2Forbit_info%2Forbit_number&page_num=1\n",
      "Order request response XML content:  b'<?xml version=\"1.0\" encoding=\"UTF-8\" standalone=\"yes\"?>\\n<eesi:agentResponse xsi:schemaLocation=\"http://eosdis.nasa.gov/esi/rsp/e https://newsroom.gsfc.nasa.gov/esi/8.1/schemas/ESIAgentResponseExternal.xsd\" xmlns=\"\" xmlns:iesi=\"http://eosdis.nasa.gov/esi/rsp/i\" xmlns:ssw=\"http://newsroom.gsfc.nasa.gov/esi/rsp/ssw\" xmlns:eesi=\"http://eosdis.nasa.gov/esi/rsp/e\" xmlns:esi=\"http://eosdis.nasa.gov/esi/rsp\" xmlns:xsi=\"http://www.w3.org/2001/XMLSchema-instance\">\\n    <order>\\n        <orderId>5000000320326</orderId>\\n        <Instructions>You may receive an email about your order if you specified an EMAIL address. &lt;br/&gt;&lt;br/&gt;The instructions used to process this order are:  Bounding Shape={\"type\":\"FeatureCollection\",\"crs\":{\"type\":\"name\",\"properties\":{\"name\":\"urn:ogc:def:crs:OGC:1.3:CRS84\"}},\"features\":[{\"type\":\"Feature\",\"properties\":{\"Name\":null,\"description\":null,\"timestamp\":null,\"begin\":null,\"end\":null,\"altitudeMode\":null,\"tessellate\":-1,\"extrude\":0,\"visibility\":-1,\"drawOrder\":null,\"icon\":null},\"geometry\":{\"type\":\"Polygon\",\"coordinates\":[[[-38.4473901024492,66.9086039587665],[-38.0989491581698,66.8478161818577],[-38.0544246374648,66.7145224687654],[-38.6939296965283,66.4309364993326],[-39.231182022375,66.4596776379638],[-39.2106092081328,66.6988972341601],[-38.4473901024492,66.9086039587665]]]}}]}. Band(s)=/ancillary_data/atlas_sdp_gps_epoch,/gt1l/land_ice_segments/atl06_quality_summary,/gt1l/land_ice_segments/delta_time,/gt1l/land_ice_segments/h_li,/gt1l/land_ice_segments/h_li_sigma,/gt1l/land_ice_segments/latitude,/gt1l/land_ice_segments/longitude,/gt1l/land_ice_segments/segment_id,/gt1l/land_ice_segments/sigma_geo_h,/gt1r/land_ice_segments/atl06_quality_summary,/gt1r/land_ice_segments/delta_time,/gt1r/land_ice_segments/h_li,/gt1r/land_ice_segments/h_li_sigma,/gt1r/land_ice_segments/latitude,/gt1r/land_ice_segments/longitude,/gt1r/land_ice_segments/segment_id,/gt1r/land_ice_segments/sigma_geo_h,/gt2l/land_ice_segments/atl06_quality_summary,/gt2l/land_ice_segments/delta_time,/gt2l/land_ice_segments/h_li,/gt2l/land_ice_segments/h_li_sigma,/gt2l/land_ice_segments/latitude,/gt2l/land_ice_segments/longitude,/gt2l/land_ice_segments/segment_id,/gt2l/land_ice_segments/sigma_geo_h,/gt2r/land_ice_segments/atl06_quality_summary,/gt2r/land_ice_segments/delta_time,/gt2r/land_ice_segments/h_li,/gt2r/land_ice_segments/h_li_sigma,/gt2r/land_ice_segments/latitude,/gt2r/land_ice_segments/longitude,/gt2r/land_ice_segments/segment_id,/gt2r/land_ice_segments/sigma_geo_h,/gt3l/land_ice_segments/atl06_quality_summary,/gt3l/land_ice_segments/delta_time,/gt3l/land_ice_segments/h_li,/gt3l/land_ice_segments/h_li_sigma,/gt3l/land_ice_segments/latitude,/gt3l/land_ice_segments/longitude,/gt3l/land_ice_segments/segment_id,/gt3l/land_ice_segments/sigma_geo_h,/gt3r/land_ice_segments/atl06_quality_summary,/gt3r/land_ice_segments/delta_time,/gt3r/land_ice_segments/h_li,/gt3r/land_ice_segments/h_li_sigma,/gt3r/land_ice_segments/latitude,/gt3r/land_ice_segments/longitude,/gt3r/land_ice_segments/segment_id,/gt3r/land_ice_segments/sigma_geo_h,/orbit_info/cycle_number,/orbit_info/rgt,/orbit_info/orbit_number. Granule id(s)=SC:ATL06.001:161147902,SC:ATL06.001:161028929,SC:ATL06.001:161160247,SC:ATL06.001:161158660,SC:ATL06.001:161162283,SC:ATL06.001:161191710,SC:ATL06.001:161203947,SC:ATL06.001:161193462,SC:ATL06.001:161212824,SC:ATL06.001:161218211. Temporal search start=2018-10-22T00:00:00 end=2019-02-22T23:59:59. Email address=kpoinar@buffalo.edu. Processing tool=ICESAT2.</Instructions>\\n    </order>\\n    <contactInformation>\\n        <contactName>NSIDC User Services</contactName>\\n        <contactEmail>nsidc@nsidc.org</contactEmail>\\n    </contactInformation>\\n    <processInfo>\\n        <processDuration>PT0.113S</processDuration>\\n        <subagentId>ICESAT2</subagentId>\\n    </processInfo>\\n    <requestStatus>\\n        <status>processing</status>\\n        <numberProcessed>0</numberProcessed>\\n        <totalNumber>10</totalNumber>\\n    </requestStatus>\\n</eesi:agentResponse>\\n'\n",
      "order ID:  5000000320326\n",
      "status URL:  https://n5eil02u.ecs.nsidc.org/egi/request/5000000320326\n",
      "HTTP response from order response URL:  201\n",
      "Data request  1  is submitting...\n",
      "Initial request status is  processing\n",
      "Status is not complete. Trying again.\n",
      "Retry request status is:  complete_with_errors\n",
      "error messages:\n",
      "['161160247:NoMatchingData - No data found that matched subset constraints. '\n",
      " 'Exit code 3.',\n",
      " '161158660:NoMatchingData - No data found that matched subset constraints. '\n",
      " 'Exit code 3.',\n",
      " '161191710:NoMatchingData - No data found that matched subset constraints. '\n",
      " 'Exit code 3.',\n",
      " '161212824:NoMatchingData - No data found that matched subset constraints. '\n",
      " 'Exit code 3.',\n",
      " '161218211:NoMatchingData - No data found that matched subset constraints. '\n",
      " 'Exit code 3.',\n",
      " 'PT2.406S',\n",
      " 'ICESAT2']\n",
      "Zip download URL:  https://n5eil02u.ecs.nsidc.org/esir/5000000320326.zip\n",
      "Beginning download of zipped output...\n",
      "Data request 1 is complete.\n",
      "Order:  2\n",
      "Request HTTP response:  201\n",
      "Order request URL:  https://n5eil02u.ecs.nsidc.org/egi/request?short_name=ATL06&version=001&temporal=2018-10-22T00%3A00%3A00Z%2C2019-02-22T23%3A59%3A59Z&time=2018-10-22T00%3A00%3A00%2C2019-02-22T23%3A59%3A59&polygon=-38.4473901024492%2C66.9086039587665%2C-39.2106092081328%2C66.6988972341601%2C-39.231182022375%2C66.4596776379638%2C-38.6939296965283%2C66.4309364993326%2C-38.0544246374648%2C66.7145224687654%2C-38.0989491581698%2C66.8478161818577%2C-38.4473901024492%2C66.9086039587665&request_mode=async&page_size=10&token=EB953953-2F0A-785C-19A8-99256CBDDFDE&email=kpoinar%40buffalo.edu&Coverage=%2Fancillary_data%2Fatlas_sdp_gps_epoch%2C%2Fgt1l%2Fland_ice_segments%2Fatl06_quality_summary%2C%2Fgt1l%2Fland_ice_segments%2Fdelta_time%2C%2Fgt1l%2Fland_ice_segments%2Fh_li%2C%2Fgt1l%2Fland_ice_segments%2Fh_li_sigma%2C%2Fgt1l%2Fland_ice_segments%2Flatitude%2C%2Fgt1l%2Fland_ice_segments%2Flongitude%2C%2Fgt1l%2Fland_ice_segments%2Fsegment_id%2C%2Fgt1l%2Fland_ice_segments%2Fsigma_geo_h%2C%2Fgt1r%2Fland_ice_segments%2Fatl06_quality_summary%2C%2Fgt1r%2Fland_ice_segments%2Fdelta_time%2C%2Fgt1r%2Fland_ice_segments%2Fh_li%2C%2Fgt1r%2Fland_ice_segments%2Fh_li_sigma%2C%2Fgt1r%2Fland_ice_segments%2Flatitude%2C%2Fgt1r%2Fland_ice_segments%2Flongitude%2C%2Fgt1r%2Fland_ice_segments%2Fsegment_id%2C%2Fgt1r%2Fland_ice_segments%2Fsigma_geo_h%2C%2Fgt2l%2Fland_ice_segments%2Fatl06_quality_summary%2C%2Fgt2l%2Fland_ice_segments%2Fdelta_time%2C%2Fgt2l%2Fland_ice_segments%2Fh_li%2C%2Fgt2l%2Fland_ice_segments%2Fh_li_sigma%2C%2Fgt2l%2Fland_ice_segments%2Flatitude%2C%2Fgt2l%2Fland_ice_segments%2Flongitude%2C%2Fgt2l%2Fland_ice_segments%2Fsegment_id%2C%2Fgt2l%2Fland_ice_segments%2Fsigma_geo_h%2C%2Fgt2r%2Fland_ice_segments%2Fatl06_quality_summary%2C%2Fgt2r%2Fland_ice_segments%2Fdelta_time%2C%2Fgt2r%2Fland_ice_segments%2Fh_li%2C%2Fgt2r%2Fland_ice_segments%2Fh_li_sigma%2C%2Fgt2r%2Fland_ice_segments%2Flatitude%2C%2Fgt2r%2Fland_ice_segments%2Flongitude%2C%2Fgt2r%2Fland_ice_segments%2Fsegment_id%2C%2Fgt2r%2Fland_ice_segments%2Fsigma_geo_h%2C%2Fgt3l%2Fland_ice_segments%2Fatl06_quality_summary%2C%2Fgt3l%2Fland_ice_segments%2Fdelta_time%2C%2Fgt3l%2Fland_ice_segments%2Fh_li%2C%2Fgt3l%2Fland_ice_segments%2Fh_li_sigma%2C%2Fgt3l%2Fland_ice_segments%2Flatitude%2C%2Fgt3l%2Fland_ice_segments%2Flongitude%2C%2Fgt3l%2Fland_ice_segments%2Fsegment_id%2C%2Fgt3l%2Fland_ice_segments%2Fsigma_geo_h%2C%2Fgt3r%2Fland_ice_segments%2Fatl06_quality_summary%2C%2Fgt3r%2Fland_ice_segments%2Fdelta_time%2C%2Fgt3r%2Fland_ice_segments%2Fh_li%2C%2Fgt3r%2Fland_ice_segments%2Fh_li_sigma%2C%2Fgt3r%2Fland_ice_segments%2Flatitude%2C%2Fgt3r%2Fland_ice_segments%2Flongitude%2C%2Fgt3r%2Fland_ice_segments%2Fsegment_id%2C%2Fgt3r%2Fland_ice_segments%2Fsigma_geo_h%2C%2Forbit_info%2Fcycle_number%2C%2Forbit_info%2Frgt%2C%2Forbit_info%2Forbit_number&page_num=2\n",
      "Order request response XML content:  b'<?xml version=\"1.0\" encoding=\"UTF-8\" standalone=\"yes\"?>\\n<eesi:agentResponse xsi:schemaLocation=\"http://eosdis.nasa.gov/esi/rsp/e https://newsroom.gsfc.nasa.gov/esi/8.1/schemas/ESIAgentResponseExternal.xsd\" xmlns=\"\" xmlns:iesi=\"http://eosdis.nasa.gov/esi/rsp/i\" xmlns:ssw=\"http://newsroom.gsfc.nasa.gov/esi/rsp/ssw\" xmlns:eesi=\"http://eosdis.nasa.gov/esi/rsp/e\" xmlns:esi=\"http://eosdis.nasa.gov/esi/rsp\" xmlns:xsi=\"http://www.w3.org/2001/XMLSchema-instance\">\\n    <order>\\n        <orderId>5000000320327</orderId>\\n        <Instructions>You may receive an email about your order if you specified an EMAIL address. &lt;br/&gt;&lt;br/&gt;The instructions used to process this order are:  Bounding Shape={\"type\":\"FeatureCollection\",\"crs\":{\"type\":\"name\",\"properties\":{\"name\":\"urn:ogc:def:crs:OGC:1.3:CRS84\"}},\"features\":[{\"type\":\"Feature\",\"properties\":{\"Name\":null,\"description\":null,\"timestamp\":null,\"begin\":null,\"end\":null,\"altitudeMode\":null,\"tessellate\":-1,\"extrude\":0,\"visibility\":-1,\"drawOrder\":null,\"icon\":null},\"geometry\":{\"type\":\"Polygon\",\"coordinates\":[[[-38.4473901024492,66.9086039587665],[-38.0989491581698,66.8478161818577],[-38.0544246374648,66.7145224687654],[-38.6939296965283,66.4309364993326],[-39.231182022375,66.4596776379638],[-39.2106092081328,66.6988972341601],[-38.4473901024492,66.9086039587665]]]}}]}. Band(s)=/ancillary_data/atlas_sdp_gps_epoch,/gt1l/land_ice_segments/atl06_quality_summary,/gt1l/land_ice_segments/delta_time,/gt1l/land_ice_segments/h_li,/gt1l/land_ice_segments/h_li_sigma,/gt1l/land_ice_segments/latitude,/gt1l/land_ice_segments/longitude,/gt1l/land_ice_segments/segment_id,/gt1l/land_ice_segments/sigma_geo_h,/gt1r/land_ice_segments/atl06_quality_summary,/gt1r/land_ice_segments/delta_time,/gt1r/land_ice_segments/h_li,/gt1r/land_ice_segments/h_li_sigma,/gt1r/land_ice_segments/latitude,/gt1r/land_ice_segments/longitude,/gt1r/land_ice_segments/segment_id,/gt1r/land_ice_segments/sigma_geo_h,/gt2l/land_ice_segments/atl06_quality_summary,/gt2l/land_ice_segments/delta_time,/gt2l/land_ice_segments/h_li,/gt2l/land_ice_segments/h_li_sigma,/gt2l/land_ice_segments/latitude,/gt2l/land_ice_segments/longitude,/gt2l/land_ice_segments/segment_id,/gt2l/land_ice_segments/sigma_geo_h,/gt2r/land_ice_segments/atl06_quality_summary,/gt2r/land_ice_segments/delta_time,/gt2r/land_ice_segments/h_li,/gt2r/land_ice_segments/h_li_sigma,/gt2r/land_ice_segments/latitude,/gt2r/land_ice_segments/longitude,/gt2r/land_ice_segments/segment_id,/gt2r/land_ice_segments/sigma_geo_h,/gt3l/land_ice_segments/atl06_quality_summary,/gt3l/land_ice_segments/delta_time,/gt3l/land_ice_segments/h_li,/gt3l/land_ice_segments/h_li_sigma,/gt3l/land_ice_segments/latitude,/gt3l/land_ice_segments/longitude,/gt3l/land_ice_segments/segment_id,/gt3l/land_ice_segments/sigma_geo_h,/gt3r/land_ice_segments/atl06_quality_summary,/gt3r/land_ice_segments/delta_time,/gt3r/land_ice_segments/h_li,/gt3r/land_ice_segments/h_li_sigma,/gt3r/land_ice_segments/latitude,/gt3r/land_ice_segments/longitude,/gt3r/land_ice_segments/segment_id,/gt3r/land_ice_segments/sigma_geo_h,/orbit_info/cycle_number,/orbit_info/rgt,/orbit_info/orbit_number. Granule id(s)=SC:ATL06.001:161216954,SC:ATL06.001:161217768,SC:ATL06.001:161246963,SC:ATL06.001:161254483,SC:ATL06.001:161256438,SC:ATL06.001:161323830,SC:ATL06.001:161326966,SC:ATL06.001:161335179,SC:ATL06.001:161336641. Temporal search start=2018-10-22T00:00:00 end=2019-02-22T23:59:59. Email address=kpoinar@buffalo.edu. Processing tool=ICESAT2.</Instructions>\\n    </order>\\n    <contactInformation>\\n        <contactName>NSIDC User Services</contactName>\\n        <contactEmail>nsidc@nsidc.org</contactEmail>\\n    </contactInformation>\\n    <processInfo>\\n        <processDuration>PT0.114S</processDuration>\\n        <subagentId>ICESAT2</subagentId>\\n    </processInfo>\\n    <requestStatus>\\n        <status>processing</status>\\n        <numberProcessed>0</numberProcessed>\\n        <totalNumber>9</totalNumber>\\n    </requestStatus>\\n</eesi:agentResponse>\\n'\n",
      "order ID:  5000000320327\n",
      "status URL:  https://n5eil02u.ecs.nsidc.org/egi/request/5000000320327\n",
      "HTTP response from order response URL:  201\n",
      "Data request  2  is submitting...\n",
      "Initial request status is  processing\n",
      "Status is not complete. Trying again.\n",
      "Retry request status is:  complete_with_errors\n",
      "error messages:\n",
      "['161246963:NoMatchingData - No data found that matched subset constraints. '\n",
      " 'Exit code 3.',\n",
      " '161256438:NoMatchingData - No data found that matched subset constraints. '\n",
      " 'Exit code 3.',\n",
      " 'PT2.192S',\n",
      " 'ICESAT2']\n",
      "Zip download URL:  https://n5eil02u.ecs.nsidc.org/esir/5000000320327.zip\n",
      "Beginning download of zipped output...\n",
      "Data request 2 is complete.\n"
     ]
    }
   ],
   "source": [
    "# Finally, we'll download the data directly to this notebook directory in a new Outputs folder. \n",
    "# The progress of each order will be reported.\n",
    "\n",
    "    \n",
    "# Let's run our request loop with subsetting services applied. \n",
    "# We will post the KML file directly to the API:\n",
    "\n",
    "# Request data service for each page number, and unzip outputs\n",
    "\n",
    "for i in range(page_num):\n",
    "    page_val = i + 1\n",
    "    print('Order: ', page_val)\n",
    "    subset_params.update( {'page_num': page_val} )\n",
    "    \n",
    "# Post polygon to API endpoint for polygon subsetting to subset based on original, non-simplified KML file\n",
    "\n",
    "    shape_post = {'shapefile': open(kml_filepath, 'rb')}\n",
    "    request = session.post(base_url, params=subset_params, files=shape_post) \n",
    "    \n",
    "# FOR ALL OTHER REQUESTS THAT DO NOT UTILIZED AN UPLOADED POLYGON FILE, USE A GET REQUEST INSTEAD OF POST:\n",
    "#     request = session.get(base_url, params=request_params)\n",
    "    \n",
    "    print('Request HTTP response: ', request.status_code)\n",
    "\n",
    "# Raise bad request: Loop will stop for bad response code.\n",
    "    request.raise_for_status()\n",
    "    print('Order request URL: ', request.url)\n",
    "    esir_root = ET.fromstring(request.content)\n",
    "    print('Order request response XML content: ', request.content)\n",
    "\n",
    "# Look up order ID\n",
    "    orderlist = []   \n",
    "    for order in esir_root.findall(\"./order/\"):\n",
    "        orderlist.append(order.text)\n",
    "    orderID = orderlist[0]\n",
    "    print('order ID: ', orderID)\n",
    "\n",
    "# Create status URL\n",
    "    statusURL = base_url + '/' + orderID\n",
    "    print('status URL: ', statusURL)\n",
    "\n",
    "# Find order status\n",
    "    request_response = session.get(statusURL)    \n",
    "    print('HTTP response from order response URL: ', request_response.status_code)\n",
    "    \n",
    "# Raise bad request: Loop will stop for bad response code.\n",
    "    request_response.raise_for_status()\n",
    "    request_root = ET.fromstring(request_response.content)\n",
    "    statuslist = []\n",
    "    for status in request_root.findall(\"./requestStatus/\"):\n",
    "        statuslist.append(status.text)\n",
    "    status = statuslist[0]\n",
    "    print('Data request ', page_val, ' is submitting...')\n",
    "    print('Initial request status is ', status)\n",
    "\n",
    "# Continue to loop while request is still processing\n",
    "    while status == 'pending' or status == 'processing': \n",
    "        print('Status is not complete. Trying again.')\n",
    "        time.sleep(10)\n",
    "        loop_response = session.get(statusURL)\n",
    "\n",
    "# Raise bad request: Loop will stop for bad response code.\n",
    "        loop_response.raise_for_status()\n",
    "        loop_root = ET.fromstring(loop_response.content)\n",
    "\n",
    "# Find status\n",
    "        statuslist = []\n",
    "        for status in loop_root.findall(\"./requestStatus/\"):\n",
    "            statuslist.append(status.text)\n",
    "        status = statuslist[0]\n",
    "        print('Retry request status is: ', status)\n",
    "        if status == 'pending' or status == 'processing':\n",
    "            continue\n",
    "\n",
    "# Order can either complete, complete_with_errors, or fail:\n",
    "# Provide complete_with_errors error message:\n",
    "    if status == 'complete_with_errors' or status == 'failed':\n",
    "        messagelist = []\n",
    "        for message in loop_root.findall(\"./processInfo/\"):\n",
    "            messagelist.append(message.text)\n",
    "        print('error messages:')\n",
    "        pprint.pprint(messagelist)\n",
    "\n",
    "# Download zipped order if status is complete or complete_with_errors\n",
    "    if status == 'complete' or status == 'complete_with_errors':\n",
    "        downloadURL = 'https://n5eil02u.ecs.nsidc.org/esir/' + orderID + '.zip'\n",
    "        print('Zip download URL: ', downloadURL)\n",
    "        print('Beginning download of zipped output...')\n",
    "        zip_response = session.get(downloadURL)\n",
    "        # Raise bad request: Loop will stop for bad response code.\n",
    "        zip_response.raise_for_status()\n",
    "        with zipfile.ZipFile(io.BytesIO(zip_response.content)) as z:\n",
    "            z.extractall(path)\n",
    "        print('Data request', page_val, 'is complete.')\n",
    "    else: print('Request failed.')\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Clean up the Output folder by removing individual order folders:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 80,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "/home/jovyan/ATL06/seeps\n",
      "/home/jovyan/ATL06/seeps\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "['.nfs8ac03d86dcbc755400000002',\n",
       " 'processed_ATL06_20181026171133_04290103_001_01.h5',\n",
       " 'processed_ATL06_20181030170315_04900103_001_01.h5',\n",
       " 'processed_ATL06_20181128153912_09320103_001_01.h5',\n",
       " 'processed_ATL06_20181212033328_11380105_001_01.h5',\n",
       " 'processed_ATL06_20181216032509_11990105_001_01.h5',\n",
       " 'processed_ATL06_20181227141518_13740103_001_01.h5',\n",
       " 'processed_ATL06_20181231140702_00480203_001_01.h5',\n",
       " 'processed_ATL06_20190114020111_02540205_001_01.h5',\n",
       " 'processed_ATL06_20190125125128_04290203_001_01.h5',\n",
       " 'processed_ATL06_20190129124312_04900203_001_01.h5',\n",
       " 'processed_ATL06_20190212003725_06960205_001_01.h5',\n",
       " 'processed_ATL06_20190216002906_07570205_001_01.h5']"
      ]
     },
     "execution_count": 80,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "print(path)\n",
    "\n",
    "#Clean up Outputs folder by removing individual granule folders \n",
    "\n",
    "for root, dirs, files in os.walk(path, topdown=False):\n",
    "    for file in files:\n",
    "        try:\n",
    "            shutil.move(os.path.join(root, file), path)\n",
    "        except OSError:\n",
    "            pass\n",
    "        \n",
    "for root, dirs, files in os.walk(path):\n",
    "    for name in dirs:\n",
    "        os.rmdir(os.path.join(root, name))\n",
    "        \n",
    "#List files\n",
    "print(path)\n",
    "\n",
    "sorted(os.listdir(path))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.7"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}
